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Abstract 

Interest on the part of European Institutions to explore the potential of MOOCs for 
providing the kind of inclusive and multicultural education that was needed for 21st 
century learners was confirmed by funding of projects such as EMMA 
(http://www.europeanmoocs.eu). The definition of "European" for the EMMA consortium 
was based on diversity - of culture, language and approach - and integration of the 
same. This paper looks at how EMMA is responding to the challenge of embracing 
diversity and integration in the same technological artefact through the use of 
embedded translation services, and an experimental PLE. We will also introduce the ad 
hoc evaluation services that are designed to test the impact of the EMMA concept. 
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1. Introduction 

In a special issue of the International Review of Education (Alidou et. al 2011) the 
editors state that quality multilingual education is a political and technical response to 
the educational rights and requirements of learners in the developed and developing 
world. Multilingual and multicultural skills are viewed as a communicative proficiency 
that is necessary for people to fully function in the 21st century, and language is 
obviously the main vehicle for the expression of culture. 

The Unesco position paper, "Education in a Multilingual World", (2003) states its support 
for mother tongue instruction to improve the knowledge and experience of the learners 
and teacher; bilingual and/or multilingual education at all levels of education as a 
means of promoting both social and gender equality; and supports language-learning as 
an essential component of inter-cultural education to encourage understanding between 
different population groups and ensure respect for fundamental rights. 

European Institutions have also produced papers and events like the 2014 European 
conference: "Why languages matter: European and national perspectives on 
multilingualism" (1) to promote policies to encourage multilingualism in parallel with 
policies to modernise HE and create a pan-European approach with comparable, 
compatible and coherent systems across the continent. 

MOOCs initiatives have mushroomed over the last few years and recent figures from 
Class Central indicate that 35 million students signed up to a MOOC in 2015, and that 
500 universities offered 4,200 courses. One of the reasons for this explosion was the 
desire to democratise education and to make high-level learning content freely available 
as a tool for social growth and inclusion and respond to global needs for qualified digital 
and mobile citizens. 
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A recent survey shows that although MOOCs are sometimes delivered in national 
languages in Europe, the predominant language of delivery remains English and 
interestingly, this has not been an issue of major debate, and has not raised doubt 
about the effectiveness of global knowledge dissemination (Gaebel 2013). And where 
delivery is in the national language, the policy has been to deliver only in this language. 
This is true of F.U.N, the French government-led MOOC platform launched in 2013, and 
MiriadaX, the Spanish platform. In these cases the language can become the defining 
feature, with platform use expanding outside national boundaries but "geo-linguistically" 
rather than geo-politically, into francophone Africa and Spanish-speaking Latin America. 
Although Stephen Downes pointed out (Downes 2014) the desirability of including 
diverse linguistic cultures within any course delivery and learning community, few 
MOOC initiatives after his own actually seemed to provide this. 

For EMMA (2) the major challenge was how to embrace diversity - of language, 
approach and culture - within a single technological artifact and how to propose an 
exploitation approach that would encourage comparative study and effective interaction 
between learners from diverse language groups in the one community. This paper aims 
to explore how the EMMA consortium responded to this challenge by experimenting with 
the creation of inbuilt translation systems and PLE tools. And describes the series of 
tracking and survey tools that were designed to evaluate MOOC provider and learner 
response to their proposal. 

2. Multiculturalism through translation 

If EMMA wanted to preserve linguistic diversity and offer a variety of learning content 
from different HE Institutions in Europe via mother tongue instruction, then the only 
way that access could be broadened to a wider audience was through the provision of 
translations. Human translation is too expensive, crowd-sourcing unpredictable and 
Google incompatible, so The Universitat Politecnica de Valencia, experts in machine 
translation system development, became the partner responsible for tackling this 
linguistic challenge. Their method was to develop ad hoc systems for the automatic 
speech recognition (ASR) and transcription and machine translation (MT) of the 7 
partner languages into English (Catalan, Dutch, Estonian, French, Italian, Portuguese, 
Spanish) and also Polish. This meant that both video and textual courseware could be 
produced in these language combinations. UPV methodology involved training the 
machines using large amounts of in-domain parallel text and audio materials. Partners 
edited initial transcripts and translations thus providing further input for the machines to 
improve subsequent performance. Human editing of machine translations on new 
courses or revised editions of other MOOCs are also part of the feedback loop and help 
to account for continued improvement as the project goes on. UPV also developed 2 
systems from English into Italian and Spanish in the same way. English is then used as 
a bridging language to produce three and four-language MOOCs as a first step towards 
multilingual delivery of MOOCs. The EMMA interface and signposting language is also 
available in multilingual version. In terms of human editing, EMMA partners have 
identified syntax, inflexions, word order in questions and proper nouns as problematic. 

If we take quality measures like Word Error Rate for transcription, and Bleu (3) for 
translation the EMMA automatic systems are currently performing well when compared 
with similar services e.g. Google, as shown in the tables below. Dutch is a difficult 
language to transcribe because of articles, word order and modal particles and produces 
error rates of 41% on Google transcriptions for Youtube, whereas this reduces to 24.5% 
on EMMA (Brouns et al. 2015). Similarly, Portuguese is also considered difficult because 
of word stress and certain phonemes and Brazilian and Portuguese varieties but 
achieves 55.5% match with human translation on EMMA system compared to 45.4% on 
Google. 


54 




The EUROCALL Review, Volume 23, No. 2, September 2015 


The tables also show how EMMA systems are themselves improving over time. 
Translation of text and video, taking into consideration the human editing work 
involved, is currently reduced by 50% on EMMA. 


Language 

EMMA 

M10 -+ M22 

YouTube 

Languages 

EMMA 

M10 -+ M22 

Google 

Spanish 

15.9 

14.8 

22.5 (+ 52%) 

Pt >En 

52.8 

55.5 

45.4 (-18%) 

Italian 

21.2 

17.1 

31.6 (+ 85%) 

It—+En 

42.9 

47.8 

37.5 (- 22%) 

French 


20.6 

32.0 (+ 55%) 

NI->En 

38.0 

41.6 

34.0 (-18%) 

Dutch 

27.5 

24.5 

41.1 (+ 67%) 

Es-»En 

Fr-*En 

40.9 

40.9 

33.1 

34.6 (-15%) 

30.8 (- 7%) 

Estonian 

— 

27.1 

— 

Et+En 


27.8 

26.3 (• 5%) 

English 

41.4 

39.2 

65.9 (+41%) 

En->lt 

_ 

52.6 

44.1 (-16%) 

Portuguese 

45.9 

43.0 

62.3 (+ 31%) 

En-»Es 

42.5 

42.5 

38.3 (-10%) 


Figure 1. Results on transcription quality. 


Figure 2. Results on translation quality. 


It is not only the quality of the translation system that has improved over time. UPV 
produced an attractive and easy to use interface with parallel editing windows as shown 
in the image below for both transcription and translation. Advanced features allow for 
segmentation of the subtitles to ensure synchrony. EMMA has a seamless interface with 
UPV enabling MOOC authors to do all their transcription and translation editing work as 
they author their courses. 
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Figure 3. EMMA and UPV system. 

Experimentation with three and four-language MOOCs is under way, despite the time 
commitment required to provide native speaker translation revisions in each of the 
target languages. Ethnographic data from expectations questionnaires provides 
information about student language competencies and interest, which can be 
triangulated with analytics data about which delivery language is chosen, and whether 
and where translations are accessed. 

Recent data regarding a course from the Open University of Netherlands has shown that 
75% of students studied the MOOC in a single language, whereas 25% preferred to use 
a mix of both languages. 
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■ Fully in English ■ Fully in Dutch 


Partly in English and partly in Dutch 



Figure 4. Dutch language. 

However, MOOC learning is also a social activity and without learning communities it is 
impossible to exploit their massive dimension (Siemens 2005). EMMA and UPV are 
responding to this challenge by accelerating translation time to almost real time, so that 
the same web services could be used by learners in the conversations and blogs to 
access comments in other languages and create multicultural environments. 

3. Multiculturalism through customization 

In a recent contribution to the EMMA newsletter, Stephen Downes writes about the 
importance of personal rather than personalised learning, where the role of the 
educational system is not to provide learning, it is to support learning. Meanwhile, the 
decisions about what to learn, how to learn, and where to learn are made outside the 
educational system, and principally, by the individual learners themselves. 

Personal learning often begins informally, on an ad hoc basis, driven by the need to 
complete some task or achieve some objective. The learning is a means to an end, 
rather than the end in itself. EMMA wanted to encourage learners to interact with a 
variety of MOOCs on the platform and widen cultural perspectives through studying a 
course produced in another language, or comparing similar topics but treated in 
different ways by the Universities offering them. In order to do this, EMMA devised a 
series of editing tools. Users can create a customised learning path by selecting and 
harvesting different learning objects from various MOOCs and using them as building 
blocks, which they can then integrate with other authoritative resources from the web. 
These are saved in a coursebook in the PLE, where learners can reorganise the 
materials and add their own study notes. Personal adaptation of the learning content in 
this way is one of the metrics for evaluation of the EMMA project, to see how often and 
what type of coursebooks are created. To render this building block approach more 
effective, MOOC courses were sourced in clusters according to discipline or target 
audience (De Rosa et al. 2014). 

It is essential to have a significant variety of topics on board to encourage this 
personalised and/or multicultural approach. MOOC sourcing activity has been intensive 
of late and EMMA is still inviting HE Institutions to join the team and host their MOOC on 
the platform. 

4. Multiculturalism through pedagogy 

Recent learning analytics data shows that the platform accommodates a range of 
instructional design: 

• iMOOCs from Open University of Portugal whereby learning content is not 
predetermined and evolves through interaction between participants, teachers and 
materials; 
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• cMOOCs from University of Bourgogne, where learning is enhanced through the 
creation of a learning community and communication on a variety of social 
networks; 

• MOOCs used as part of on-campus blended learning courses at University of 
Naples, Federico II in a flipped classroom approach; 

• Hybrid MOOCs from University of Urbino, delivered in streaming with live chat; 

• xMOOCs from University Polytechnic of Valencia with lecture-capture and 
authoritative learning content. 

5. Conclusions 

We have set out to show how we took the concept of multiculturalism that was central 
to our educational offer and split it up into its three constituent components. We defined 
the three drivers of diversity as approach, language and personalisation (i.e. the 
possibility to bring one's own culture into the learning experience), and we looked at 
how we could bring the three dimensions together in the one learning environment. 
Unique of its kind, EMMA is now in deployment phase and gaining recognition in Europe. 
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Notes 

[1] http://ec.europa.eu/languages/inspire/1009-florence_en.htm. 

[2] The project is headed by University of Naples Federico II, leaders in the field of OER 
with a awarded project Federica (http://www.federica.unina.it/), now also with a MOOC 
offer (http://www.federica.eu). 11 different organisations from 7 European countries 
are involved in the EMMA Project: Universita degli Studi di Napoli Federico II, Italy 
(Contractor); ATOS, Spain; IPSOS srl, Italy; Universitat Oberta de Catalunya, Spain; 
Open Universiteit, Netherlands; Universidade Aberta, Portugal; Universite de 
Bourgogne, France; Tallinn University, Estonia; Universitat Politecnica de Valencia, 
Spain; CSP - Innovazione nelle ICT S.C.A R.L, Italy; ATiT, Belgium. EMMA has received 
funding from the European Union's Competitiveness and Innovation framework 
Programme CIP 2007-2013, under grant agreement n. 621030. 

[3] Bleu: Bilingual evaluation understudy. An evaluation system comparing machine 
translation word order and choice with human translation. 


58 





